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Abstract 

This  paper  documents  the  results  of  a  multi-year  Army  Materiel  Command-sponsored 
research  project  which  employed  a  structured  case  study  approach  to  examine  the  history  and 
processes  that  had  resulted  in  the  introduction  of  a  number  of  technology-based  Army  systems 
in  time  to  make  a  positive  contribution  to  the  outcome  of  Desert  Storm.  In  addition  to  the  fifteen 
case  studies  documenting  these  programs,  a  common  set  of  data  was  obtained  for  each  system 
studied.  These  data  were  analyzed  to  identify  factors  contributing  to  successful  systems 
development;  this  paper  contains  the  results  of  this  analysis. 
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Several  of  the  statistically  significant  relationships  found  involve  factors  that  are  related 
to  the  stability  of  the  program.  When  key  members  of  the  project  team  left  the  program  too 
early,  project  outcome  suffered.  Further,  both  project  funding  cutbacks  and  project  team  turn¬ 
over  negatively  correlated  with  the  quality  of  the  testing  program  and  the  timeliness  of  key  test 
events.  These  two  attributes  of  the  testing  program  also  had  the  strongest  correlation  with 
project  outcomes.  In  addition,  changes  in  systems  requirements  during  development  correlated 
with  poor  project  cost  performance.  Finally,  turn-over  in  key  user-representative  personnel 
correlated  negatively  with  system  performance  in  the  field.  A  central  conclusion  from  this  study 
is  that  shorter  development  cycle-times  favorably  correlate  with  key  project  outcome  variables, 
largely  by  minimizing  the  exposure  of  the  project  to  destabilizing  influences  which  were  also 
shown  to  correlate  negatively  with  these  same  outcome  variables. 

Keywords:  technology-based  Army  systems,  project  outcome,  system  requirements, 
development  cycle-time 

Introduction 

This  paper  documents  the  results  of  a  research  project  of  several  years’  duration  which 
employed  a  structured  case  study  approach  to  examine  the  history  and  processes  that  had 
resulted  in  the  introduction  of  a  number  of  technology-based  Army  systems  in  time  to  make  a 
positive  contribution  to  the  outcome  of  Desert  Storm.  The  15  case  studies  that  resulted  were 
developed  on  systems  ranging  from  the  M829A1  “silver  bullet”  to  the  GUARDRAIL  Common 
Sensor  and  the  APACHE  attack  helicopter. 


Research  Project  Information 


•  Principal  Sponsor:  Army  Material  Command 

•  Principal  Investigators:  Bill  Lucas  (MIT)  and  Dick 
Rhoades  (UAH) 

•  Research  Period:  September  1999  to  May  2004  (data 
analysis  and  report  preparation  continued  into  2005) 

•  Funding:  -$200,000 

•  Research  Purpose:  Examine  the  history  and  processes 
used  in  the  development  of  a  number  of  Army  systems 
which  made  a  positive  contribution  on  the  battlefield 
during  Desert  Storm 

—determine  factors  which  influence  success 
—prepare  case  studies 


Figure  1.  Project  Overview 
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Systems  Studied 


Svstem 

Researcher 

Commoditv  category 

APACHE  attack  helicopter 

Ference 

Aviation 

TADS/PNVS  (target  acquisition  and 
designation/pilot’s  night  vision  systems) 

Oelrich 

Aviation 

MLRS  rocket  system 

Sherman 

Missiles 

ATACMS  missile  system 

Romanczuk 

Missiles 

M40  chemical  protective  mask 

Ruocco 

Soldier  support 

Dismounted  microclimate  cooler 

Note:  Did  not  enter  production 

Ruocco 

Soldier  support 

Mounted  microclimate  cooler 

Ruocco 

Soldier  support 

M829-A1  armor-piercing  kinetic  energy 
tank  ammunition 

Mitchell 

Ammunition 

FOG-M  (fiber  optic  guided  missile) 

Note:  Did  not  enter  production 

Sherman 

Missiles 

T0W-2A  (Tube-launched  missile) 

Vessels 

Missiles 

AN/TAS  4  infrared  night  sight 

Granone 

Target  acquisition 

Joint  Stars  Ground  Station 

Sherman 

Intelligence 

Guardrail  common  sensor 

Sherman 

Intelligence 

PAC-2  (PATRIOT  anti-missile  system) 

Sherman 

Missiles 

HELLFIRE  missile  system 

Johansen 

Missiles 

Figure  2.  Systems  Studied 

The  case  studies  were  developed  through  the  use  of  structured  interviews  with  key 
participants  from  the  government/contractor  team  that  developed  each  system.  In  addition  to  the 
case  studies,  this  process  resulted  in  collection  of  a  common  set  of  data  for  the  systems  studied 
which  could  then  be  analyzed  to  identify  factors  contributing  to  successful  system  development. 
The  results  of  this  analysis  are  contained  in  this  paper.  Two  of  the  15  case  studies  examined 
systems  which  might  have  been  useful  on  the  battlefield  (based  on  the  views  of  Army  technical 
leaders),  but  that  failed  to  successfully  complete  development.  The  intent  of  including  failures  in 
the  research  was  to  provide  a  basis  for  distinguishing  factors  which  contributed  to  both 
successful  and  unsuccessful  system  developments.  While  they  are  useful  for  the  qualitative 
lessons  they  offer,  two  cases  are  inadequate  for  quantitative  analysis;  most  analysis  focuses  on 
the  13  successful  cases.  The  study  is,  therefore,  primarily  an  assessment  of  contributors  to  the 
relative  degree  of  success.1 


1  The  LeanTEC  project  was  a  four-year  study  of  the  development  and  transition  of  technology-dependent 
systems  in  the  aerospace  industry,  supported  by  a  cooperative  research  agreement  between  the  US  Air 
Force  Manufacturing  Technology  Office  and  The  Boeing  Company. 
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Research  Methodology 


•  Army  RDEC  and  PM  leadership  nominated  systems 
which  either  did  or  could  have  impacted  Desert  Storm 

•  Researchers  (intended  to  be  “free”  Army  student  labor) 
selected  a  system  from  list  of  candidates 

•  “Structured  thesis”  approach  used  to  gather  comparable 
data  on  each  system  studied,  but  allow  researcher  to 
document  areas  of  particular  interest  in  each  case  study 

•  Modified  version  of  questionnaire  used  on  LeanTEC* 
was  administered  to  Army  and  contractor  development 
team  members;  researcher  integrated  responses 
--produced  composite  “best  answer”  questionnaire 
—produced  case  study  on  system  development 


15  systems,  13  produced  dictated  a  focus  on  relative  success  factors 


Figure  3.  Methodology  Employed 

The  heart  of  any  systematic  study  is  the  definition  of  a  common  outcome  measure  that 
allows  comparison.  The  obvious  path  was  to  compare  the  projects  and  systems  based  on  their 
performance  relative  to  their  agreed-upon  goals  and  requirements.  Each  project  had  a  budget, 
a  systems  procurement  cost  goal,  a  set  of  technical  requirements,  and  completion  dates.  In 
addition,  three  questions  of  performance  are  immediately  observable  and  easily  remembered  by 
project  managers:  Did  the  system  go  into  production?  Once  production  was  started,  were 
problems  found  that  required  that  further  engineering  changes  be  made?  And  did  the  system 
perform  well  in  its  use  in  Desert  Storm?  Structured  questions  were  used  to  ask  the  key 
government  and  industry  interviewees  about  how  well  their  projects  performed  in  these  areas, 
with  a  range  of  answers  that  characterized  how  badly  the  projects  had  missed  meeting  their 
objectives  if  they  had  not  been  completely  successful.  Each  of  these  outcomes  is  shown 
graphically  in  the  histograms  which  follow. 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  FOR  INFORMED  CHANGE  -476- 


Outcomes-Development  Budget 


underran  exceeded  exceeded 


Budget  Performance 


Figure  4.  Development  Budget 


Outcomes-System  unit  cost 


7 

6 

5 

4 

3 

2 

1 

O 


MLRS,  ATACMS, 

HELLFIRE.TADS/PNVS,  M829A1  and 


Mounted  rhicrocooler 


Guardrail,  PAC-2,  TOW  2A, 

Night  Sight,  APACHE  and 
M40  mask 


Joint  Stars 


Met  or 
exceeded 


Game  close 
to  goal 


Cost  Re  rformance 


Far  short 
of  goal 


Figure  5.  System  Unit  Cost 
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Outcomes-Technical  performance 


3 

Z 


10 

9 

8 

7 

6 


5 


4 

3 

2 

1 

0 


Met  or  Came  close 

exceeded 


Achievement  of  Technical  Goals 


Figure  6.  Technical  Performance 


Outcomes-Delay  in  transitioning  to 
production 


8 

7 

6 

5 

4 

3 

2 

1 

O 


Vehicle  microcooler, 

GUARDRAIL,  ATACMS, 

PAC-2,  MLRS,  M829  A1, 

Niqht  siaht 

APACHE,  HELLFIRE 

Joint  Stars, 

M40  Mask 

1-6  months 
late 


7-12  months 
late 


No  delay 

Development  to  production  delay 


year  +  late 


Figure  7.  Delay  in  Transitioning  to  Production 
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Outcomes-Changes  in  production 


9 

8 

7 

6 

5 

4 

3 

2 

1 

0 


No  or  almost 
no 


Joint  Stars.  TADS/PNVS. 

TOW-2A,  PAC-2,  MLRS, 

M829  A1 ,  Night  Sight, 

GUARDRAIL, 

Mounted  micro- 

cooler 

APACHE,  M40  Mask 

Minor 


Significant 


Engineering  changes  in  production 


Figure  8.  Changes  in  Production 


Six  of  the  outcome  measures  mentioned  above  were  used  to  create  a  scale  that  scores 
the  (system)  projects  from  zero  to  six  according  to  the  number  of  key  outcomes  a  project 
achieved.  If  a  project  was  (1)  transitioned  to  production  on  time,  (2)  developed  within  budget, 
(3)  had  no  late  engineering  changes,  met  both  (4)  the  goals  for  system  unit  costs  and  (5)  its 
technical  requirements,  and  encountered  (6)  no  difficulties  when  it  was  deployed  in  the  field,  it 
was  awarded  (the  maximum)  six  points. 


Outcomes-lntegrated  Scale 


1  2  3  4  5  6 


Outcome  scale 


Scale:  Sum  of  number  of  preferred  outcomes  using  six  outcome  metrics 


Figure  9.  Integrated  Scale 
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Table  1  (next  page)  contains  summary  information  on  the  15  systems  studied.  For  each 
system,  this  table  also  contains  information  on  the  duration  of  the  development  phase  of  the 
program  and  a  summary  of  the  project  manager’s  description  of  the  most  difficult  problem 
encountered.  It  is  interesting  to  note  that  lack  of  sustained  user  support  for  the  requirement  the 
system  was  intended  to  satisfy  was  mentioned  as  the  most  difficult  problem  for  the  two  failures, 
but  user-related  issues  were  not  identified  for  any  of  the  successful  development  cases. 


Svstem/case 

Development 

duration  (months) 

PM’s  most  difficult 

problem 

Key 

outcomes 
achieved  (0- 

6 1 

APACHE  attack 
helicopter 

108 

Control  of  production 
costs;  influenced  by 
integration  plant  location 
choices 

1 

TADS/PNVS  (target 
acquisition  and 
designation/pilot’s  night 
vision  systems) 

~36 

Cost  growth  in 
development 

3 

MLRS  rocket  system 

33 

Establishing  and 
managing  four-nation 
cooperative  development 
program 

6 

ATACMS  missile  system 

37 

Key  vendor  went  out  of 
business 

6 

M40  chemical  protective 
mask 

~48 

Immaturity  of  critical 
technologies 

2 

Dismounted  microclimate 
cooler 

Note:  Did  not  enter  full 
development 

Not  applicable 

Lack  of  stable  user 
requirements  due  to 
immaturity  of  technology 

Not 

applicable 

Mounted  microclimate 
cooler 

~24 

Key  vendor  failed  to 
support  integration 
schedule 

5 

M829-A1  armor-piercing 
kinetic-energy  tank 
ammunition 

~36 

Achieving  needed 
innovation  in  system 
design 

6 

FOG-M  (fiber-optic 
guided  missile) 

Note:  Did  not  complete 
development 

Not  applicable; 

Lack  of  sustained  user 
support 

Not 

applicable 

TOW-2A  (Tube-launched 
missile) 

48 

Stability  of  threat  armor 
requirements 

3 

AN/TAS  4  infrared  night 
sight 

~24 

Selection  of  unqualified 
vendor  and  split 
management 
responsibility 

4 

n.-.i'.uMiAftr.  v  hsiu,, 
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Joint  Stars  Ground 

Station 

105 

Cost  and  schedule 
growth/delivering 
complex  software 

1 

Guardrail  common  sensor 

~24 

Complexity  of  integration 
of  mission  equipment 

3 

PAC-2  (PATRIOT  anti¬ 
missile  system) 

~52 

Early  fielding  to  meet 
SCUD  missile  threat 

2 

HELLFIRE  missile 
system 

~84 

Adversarial  relationship 
between  key  vendor  and 
prime 

3 

Table  1.  Summary  Case  Information 


Standard  statistical  analysis  procedures  appropriate  for  this  number  of  cases  and  type  of 
data  were  used  to  identify  and  evaluate  correlations  between  the  factors  studied  and  the  several 
outcome  variables,  and,  in  some  cases,  among  the  factors.  The  results  of  these  analyses  are 
summarized  in  Table  2.  The  testing/simulation  and  technological  maturity  factors  were  included 
because  of  their  identification  in  recent  Government  Accounting  Office  studies  as  key 
determinants  of  success. 


Factor 

Relationships  Found/Comments 

1.  Project  team  characteristics 
and  practices: 

— leadership 

Team  leader’s  perceived  ability  to  obtain  resources,  his/her 
breadth  of  experience  and  ability  to  resolve  technical  issues  all 
are  positively  related  to  reduced  engineering  changes  during 
production  and  to  completing  development  within  budget. 

— staffing 

Low  turnover  in  key  project  team  members  relates  positively  to 
completing  development  within  budget,  to  meeting  system  unit 
cost  targets  and  to  achieving  system  performance  objectives. 

2.  Role  of  government  S&T 
organizations 

Army  labs/centers  were  typically  actively  involved  in  both  pre¬ 
development  and  development  phases,  actively  involved  in 
both  successes  and  failures,  and  actively  involved  in  both  short 
and  long  developments. 

3.  Testing  and  simulation 
approach 

Validating  component  and  system  maturity  at  the  right  time  in 
the  program  relates  positively  to  completing  development  within 
budget,  to  meeting  system  unit  cost  targets  and  to  successful 
performance  in  the  field.  The  quality  of  the  testing  and 
simulation  conducted  relates  positively  to  reduced  engineering 
changes  during  production  and  to  meeting  system  unit  cost 
targets. 

4.  Importance  of  stability: 

— funding 

Funding  uncertainty  was  related  to  increased  turnover  in  key 
project  team  members  and  the  need  to  deal  with  changes  in 
testing  plans  and  other  project  structure  issues. 

— system  requirements 

Changes  in  system  requirements,  particularly  during  the  middle 
of  development,  relate  to  an  increase  in  late  engineering 
changes  and  negatively  to  project  success  in  meeting  its  goals 
for  systems  costs. 

— key  user  (TRADOC) 

Changes  in  key  TRADOC  personnel  during  development 
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personnel 

relates  to  less  successful  performance  in  the  field. 

5.  Timely  communication  of 
problems 

Nearly  all  cases  described  timely  communication  of  problems 
from  contractor  to  government  PM  and  from  government  PM  to 
Army  leadership. 

6.  Importance  of  technology 
maturity  (TRLs) 

Maturity  of  critical  technologies  used  in  systems  studied,  as 
measured  by  TRLs,  was  similar  to  that  found  in  previous 

LeanTec  study  of  small  electronics  projects.  No  positive 
correlation  found  between  higher  TRLs  at  the  start  of 
development  and  most  outcome  variables. 

Table  2.  Summary  of  significant  relationships 


Several  of  the  statistically  significant  relationships  involve  factors  that  are  related  to  the 
stability  of  the  program.  When  key  members  of  the  project  team  left  the  program  too  early, 
project  outcome  suffered.  Further,  both  project  funding  cutbacks  and  project  team  turn-over 
negatively  correlated  with  the  quality  of  the  testing  program  and  the  timeliness  of  key  test 
events.  These  two  attributes  of  the  testing  program  also  had  the  strongest  correlation  with 
project  outcomes.  In  addition,  changes  in  systems  requirements  during  development  correlated 
with  poor  project  cost  performance.  Finally,  turn-over  in  key  user-representative  personnel 
correlated  negatively  with  system  performance  in  the  field. 


Destabilizing  Influences 


Variable 

Timing  Implications 

1.  Reductions  in  project 
funding 

Potential  for  change  in  administration  every  48  months;  typical 
turn-over  in  key  military  leaders  occurs  every  24-36  months. 
Potential  change  in  key  Congress  positions  every  24  months; 
likelihood  increases  with  development  duration 

2.  Uncertainty  in  project 
funding 

Potential  for  change  in  administration  every  48  months;  typical 
turn-over  in  key  military  leaders  occurs  every  24-36  months.  . 
Potential  change  in  key  Congress  positions  every  24  months; 
likelihood  increases  with  development  duration. 

3.  Change  in  system 
requirements 

Changes  in  the  threat  environment  occur  unpredictably,  but 
become  more  likely  with  longer  development  durations. 

Changes  in  doctrine  and  system  requirements  follow  a  similar 
pattern. 

4.  Change  in  key  user 
representatives 

Typical  turn-over  in  such  key  military  positions  occurs  every 
~36  months 

5.  Change  in  key  project  team 
members 

Typical  turn-over  in  military  acquisition  positions  occurs  every 
~36  months.  Longer  development  durations  present  more 
opportunities  for  career  moves  on  the  part  of  key  civilian  team 
members 

Figure  10.  Destabilizing  Influences 

Taken  together,  these  several  relationships  strongly  suggest  that  stability  of  program 
resources  and  objectives  is  a  very  powerful  influence  on  the  relative  success  of  the  project.  In 
reflecting  on  this  array  of  instabilities  that  could  impact  a  system  development,  it  became  clear 
that  they  had  at  least  one  thing  in  common:  The  longer  a  system  stayed  in  development,  the 
greater  chance  it  had  to  experience  one  or  more  of  these  program  destabilizing  events.  Or, 
stated  another  way,  shorter  system  development  cycles  should  result  in  better  project 
outcomes.  When  this  hypothesis  was  tested  by  examining  the  correlation  between  the  system 
development  durations  and  the  aggregate  outcome  scale  (See  the  data  in  Table  1),  a  strong 
correlation  was  found.  A  central  conclusion  from  this  study  is,  therefore,  that  shorter 
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development  cycle-times  favorably  correlate  with  key  project  outcome  variables — largely  by 
minimizing  the  exposure  of  the  project  to  destabilizing  influences  which  have  also  been  shown 
to  correlate  negatively  with  these  same  outcome  variables. 


Central  Conclusion 


Shorter  development  cycle  times  favorably 
correlate  with  key  project  outcome  variables, 
largely  by  minimizing  the  exposure  of  the  project 
to  destabilizing  influences 


Length  of  Project  Development  and  Project  Performance 

(Average  number  of  successful  outcomes) 

Three  years 

Over  3  years  or  less  Sig.  at 
Length  of  development  2.00  4.71  .002 


Figure  11.  Conclusion 

Whether  or  not  a  change  to  selecting  projects  with  shorter  development  times  is  made, 
the  Army  could  do  more  to  stabilize  the  guidance  and  resources  given  to  both  shorter  and 
longer  development  projects.  Acting  alone,  the  Army  could  do  more  to  map  rotating  personnel 
assignments  and  other  sources  of  TRADOC  change  to  project  development  cycles.  Since  it 
appears,  as  is  widely  believed,  that  changes  in  systems  requirements  made  once  projects  move 
beyond  early  development  will  almost  certainly  hurt  project  performance,  the  Army  could 
eliminate  all  but  the  most  critically  important  of  such  changes.  Both  through  contract  language 
and  informal  management  practices,  the  Army  could  work  with  its  contractors  to  provide  better 
continuity  of  development  project  staffing. 

The  defense  acquisition  community  has  long  recognized  that  lengthy  systems 
development  times  are  disadvantageous.  Sometimes  the  associated  negatives  have  been 
phrased  in  program  instability  terms;  this  study  certainly  provides  strong  empirical  support  for 
those  who  hold  these  beliefs.  Over  the  years,  a  number  of  initiatives  have  been  attempted  to 
shorten  development  cycles,  with  limited  success  where  complex  systems  were  involved.  The 
current  approach  is  referred  to  as  “spiral  development”;  its  basic  concept  is  to  get  a  useful,  if 
limited,  capability  in  the  field  quickly  and  then  introduce  additional  technology-based  capabilities 
through  further  “spirals”  of  development.  This  approach  appears  to  be  in  keeping  with  the 
implications  of  this  study’s  central  conclusion. 
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2003  -  2006  Sponsored  Acquisition  Research  Topics 

Acquisition  Management 

■  Software  Requirements  for  OA 

■  Managing  Services  Supply  Chain 

■  Acquiring  Combat  Capability  via  Public-Private  Partnerships  (PPPs) 

■  Knowledge  Value  Added  (KVA)  +  Real  Options  (RO)  Applied  to  Shipyard 
Planning  Processes 

■  Portfolio  Optimization  via  KVA  +  RO 

■  MOSA  Contracting  Implications 

■  Strategy  for  Defense  Acquisition  Research 

■  Spiral  Development 

■  BCA:  Contractor  vs.  Organic  Growth 
Contract  Management 

■  USAF  IT  Commodity  Council 

■  Contractors  in  21st  Century  Combat  Zone 

■  Joint  Contingency  Contracting 

■  Navy  Contract  Writing  Guide 

■  Commodity  Sourcing  Strategies 

■  Past  Performance  in  Source  Selection 

■  USMC  Contingency  Contracting 

■  Transforming  DoD  Contract  Closeout 

■  Model  for  Optimizing  Contingency  Contracting  Planning  and  Execution 
Financial  Management 

■  PPPs  and  Government  Financing 

■  Energy  Saving  Contracts/DoD  Mobile  Assets 

■  Capital  Budgeting  for  DoD 

■  Financing  DoD  Budget  via  PPPs 

■  ROI  of  Information  Warfare  Systems 

■  Acquisitions  via  leasing:  MPS  case 

■  Special  Termination  Liability  in  MDAPs 
Logistics  Management 

■  R-TOC  Aegis  Microwave  Power  Tubes 

■  Privatization-NOSL/NAWCI 

■  Army  LOG  MOD 
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PBL  (4) 

Contractors  Supporting  Military  Operations 
RFID  (4) 

Strategic  Sourcing 

ASDS  Product  Support  Analysis 

Analysis  of  LAV  Depot  Maintenance 

Diffusion/Variability  on  Vendor  Performance  Evaluation 

Optimizing  CIWS  Lifecycle  Support  (LCS) 


Program  Management 


Building  Collaborative  Capacity 

Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 
KVA  Applied  to  Aegis  and  SSDS 

Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module  Acquisition 

Terminating  Your  Own  Program 

Collaborative  IT  Tools  Leveraging  Competence 


A  complete  listing  and  electronic  copies  of  published  research  within  the  Acquisition 
Research  Program  are  available  on  our  website:  www.acquisitionresearch.org 
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LESSONS  FROM  ARMY  SYSTEM 

DEVELOPMENTS 


A  Presentation 
For 

The  4th  Annual  Acquisition  Research 

Symposium 


Research  Project  Information 


•  Principal  Sponsor:  Army  Material  Command 

•  Principal  Investigators:  Bill  Lucas  (MIT)  and  Dick 
Rhoades  (UAH) 

•  Research  Period:  September  1999  to  May  2004  (data 
analysis  and  report  preparation  continued  into  2005) 

•  Funding:  -$200,000 

•  Research  Purpose:  Examine  the  history  and  processes 
used  in  the  development  of  a  number  of  Army  systems 
which  made  a  positive  contribution  on  the  battlefield 
during  Desert  Storm 

--determine  factors  which  influence  success 
--prepare  case  studies 


Systems  Studied 


System 

Researcher 

Commodity  category 

APACHE  attack  helicopter 

Ference 

Aviation 

TADS/PNVS  (target  acquisition  and 
designation/pilot’s  night  vision  systems) 

Oelrich 

Aviation 

MLRS  rocket  system 

Sherman 

Missiles 

ATACMS  missile  system 

Romanczuk 

Missiles 

M40  chemical  protective  mask 

Ruocco 

Soldier  support 

Dismounted  microclimate  cooler 

Note:  Did  not  enter  production 

Ruocco 

Soldier  support 

Mounted  microclimate  cooler 

Ruocco 

Soldier  support 

M829-A1  armor-piercing  kinetic  energy 
tank  ammunition 

Mitchell 

Ammunition 

FOG-M  (fiber  optic  guided  missile) 

Note:  Did  not  enter  production 

Sherman 

Missiles 

TOW-2A  (Tube-launched  missile) 

Vessels 

Missiles 

AN/TAS  4  infrared  night  sight 

Granone 

Target  acquisition 

Joint  Stars  Ground  Station 

Sherman 

Intelligence 

Guardrail  common  sensor 

Sherman 

Intelligence 

PAC-2  (PATRIOT  anti-missile  system) 

Sherman 

Missiles 

HELLFIRE  missile  system 

Johansen 

Missiles 

Research  Methodology 


•  Army  RDEC  and  PM  leadership  nominated  systems 
which  either  did  or  could  have  impacted  Desert  Storm 

•  Researchers  (intended  to  be  “free”  Army  student  labor) 
selected  a  system  from  list  of  candidates 

•  “Structured  thesis”  approach  used  to  gather  comparable 
data  on  each  system  studied,  but  allow  researcher  to 
document  areas  of  particular  interest  in  each  case  study 

•  Modified  version  of  questionnaire  used  on  LeanTEC  was 
administered  to  Army  and  contractor  development  team 
members;  researcher  integrated  responses 

— produced  composite  “best  answer”  questionnaire 
— produced  case  study  on  system  development 


15  systems,  13  produced  dictated  a  focus  on  relative  success  factors 


Outcomes-Development  Budget 


d 


underran  exceeded  exceeded 


Budget  Performance 


5 


Outcomes-System  unit  cost 


7 

6 

5 

4 

3 

2 

1 

O 


MLRS,  ATAOMS, 

HELLFIRE,TADS/PNVS,  M829  A1  and 


Mounted  nicrocooler 


Guardrail,  PAC-2,  TOW  2A, 
Night  Sight,  APACHE  and 
M40  mask 


Joint  Stars 


Met  or 
exceeded 


Came  close 
to  goal 


Far  short 
of  goal 


Cost  Performance 


6 


Outcomes-Technical  performance 


10 

9 


2 

1 

0 


MLRS,  TOW  2- A,  ATACMS,  Night  Sight, 

APACHE,  M829A1,  HELLFIRE,  Mounted 

microcooler  and  M40  mask 

Guardrail,  Joint  Stars, 

PAC-2  and  TADS/PNVS 

Met  or  Came  close 

exceeded 


Achievement  of  Technical  Goals 


7 


Outcomes-Delay  in  transitioning  to 

production 


8 

7 

6 

5 

4 

3 

2 

1 

O 


Vehicle  microcooler, 

GUARDRAIL,  ATACMS, 

PAC-2,  MLRS,  M829A1, 

Niaht  siaht 

APACHE,  HELLFIRE 

Joint  Stars, 

M40  Mask 

No  delay  1-6  months  7-12  months  year  +  late 

late  late 


Development  to  production  delay 


8 


Outcomes-Changes  in  production 


s 

■5 

2 

E 

3 


9 

8 

7 

6 

5 

4 

3 

2 

1 

0 


. Inint  .Stars  TADS/PNVS 

TOW-2  A,  PAC-2,  MLRS, 

M829  A1 ,  Night  Sight, 

HFI  1  FIRF 

GUARDRAIL, 

A  1  > A  CMS , 

Mounted  micro- 

cooler 

APACHE,  M40  Mask 

No  or  almost 
no 


Minor 


Significant 


Engineering  changes  in  production 
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Outcomes-Operational  Performance 
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w 
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2 

1 
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Guardrail,  Joint  Stars, 

PAC-?  APACHFanri 

Ml  RS  TOW- 9  A  TAD.q/PNV.q 

HELLFIRE 

ATACMS,  M40  Mask  and  Mounted 
mirror  ooler 

Night  Sight  and 

M829  A 1 

Minor  Met  Exceeded 

problems  expectations  expectations 

Outcome 


10 


Outcomes-lntegrated  Scale 
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TADS/PNVS,  T0W2A,  Guardrail  and  HELLFIRE 


Joint  Stars  PAC-2  and 

and  APACHE  M40  Mask 

MLRS,  M829  A1  and  ATACMS 


Me 

NightSight  mi 

unted 

r.rnm 

pier 

3  4 

Outcome  scale 


Scale:  Sum  of  number  of  preferred  outcomes  using  six  outcome  metrics 


Summary  Case  Information 


Svstem/case 

Development 
duration  (months) 

PM’s  most  difficult  problem 

Kev  outcomes 

achieved  (0-6) 

APACHE  attack  helicopter 

108 

Control  of  production  costs; 
influenced  by  integration 
plant  location  choices 

1 

TADS/PNVS  (target 
acquisition  and 
designation/pilot’s  night 
vision  systems) 

~36 

Cost  growth  in  development 

3 

MLRS  rocket  system 

33 

Establishing  and  managing 
four  nation  cooperative 
development  program 

6 

ATACMS  missile  system 

37 

Key  vendor  went  out  of 
business 

6 

M40  chemical  protective 
mask 

~48 

Immaturity  of  critical 
technologies 

2 

Dismounted  microclimate 
cooler 

Note:  Did  not  enter  full 
development 

Not  applicable 

Lack  of  stable  user 
requirements  due  to 
immaturity  of  technology 

Not  applicable 

Mounted  microclimate 
cooler 

~24 

Key  vendor  failed  to  support 
integration  schedule 

5 

12 


Summary  Case  Information  (cont.) 


System/case 

Development 
Duration  (months) 

PM’s  most  difficult  problem 

Kev  outcomes 

achieved  (0-6) 

M829-A1  annor-piercing 
kinetic  energy  tank 
ammunition 

~36 

Achieving  needed 
innovation  in  system  design 

6 

FOG-M  (fiber  optic  guided 
missile) 

Note:  Did  not  complete 
development 

Not  applicable; 

Lack  of  sustained  user 
support 

Not  applicable 

TOW-2A  (Tube-launched 
missile) 

48 

Stability  of  threat  armor 
requirements 

3 

AN/TAS  4  infrared  night 
sight 

~24 

Selection  of  unqualified 
vendor  and  split 
management  responsibility 

4 

Joint  Stars  Ground  Station 

105 

Cost  and  schedule 
growth/delivering  complex 
software 

1 

Guardrail  common  sensor 

~24 

Complexity  of  integration  of 
mission  equipment 

3 

PAC-2  (PATRIOT  anti¬ 
missile  system) 

~52 

Early  fielding  to  meet 
SCUD  missile  threat 

2 

HELLFIRE  missile  system 

~84 

Adversarial  relationship 
between  key  vendor  and 
prime 

3 
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Significant  Relationships 


Factor 

Relationships  Found/Comments 

1.  Project  team  characteristics 
and  practices: 

--leadership 

Team  leader’s  perceived  ability  to  obtain  resources,  his/her  breadth  of 
experience  and  ability  to  resolve  technical  issues  all  are  positively 
related  to  reduced  engineering  changes  during  production  and 
completing  development  within  budget. 

-staffing 

Low  turnover  in  key  project  team  members  relates  positively  to 
completing  development  within  budget,  to  meeting  system  unit  cost 
targets  and  to  achieving  system  performance  objectives. 

2.  Role  of  government  S&T 
organizations 

Army  labs/centers  were  typically  actively  involved  in  both  pre¬ 
development  and  development  phases;  actively  involved  in  both 
successes  and  failures;  and  actively  involved  in  both  short  and  long 
developments. 

3.  Testing  and  simulation 
approach 

Validating  component  and  system  maturity  at  the  right  time  in  the 
program  relates  positively  to  completing  development  within  budget, 
to  meeting  system  unit  cost  targets  and  to  successful  performance  in 
the  field.  The  quality  of  the  testing  and  simulation  conducted  relates 
positively  to  reduced  engineering  changes  during  production  and  to 
meeting  system  unit  cost  targets. 
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Significant  Relationships  (continued) 


Factor 

Relationships  Found/Comments 

4.  Importance  of  stability: 

--funding 

Funding  uncertainty  was  related  to  increased  turnover  in  key  project 
team  members  and  the  need  to  deal  with  changes  in  testing  plans  and 
other  project  structure  issues. 

-system  requirements 

Changes  in  system  requirements,  particularly  during  the  middle  of 
development,  relate  to  an  increase  in  late  engineering  changes  and 
negatively  to  project  success  in  meeting  its  goals  for  systems  costs. 

-key  user  (TRADOC) 
personnel 

Changes  in  key  TRADOC  personnel  during  development  relates  to 
less  successful  performance  in  the  field. 

5.  Timely  communication  of 
problems 

Nearly  all  cases  described  timely  communication  of  problems  from 
contractor  to  government  PM  and  from  government  PM  to  Army 
leadership. 

6.  Importance  of  technology 
maturity  (TRLs) 

Maturity  of  critical  technologies  used  in  systems  studied,  as  measured 
by  TRLs,  was  similar  to  that  found  in  previous  LeanTec  study  of 
small  electronics  projects.  No  positive  correlation  found  between 
higher  TRLs  at  the  start  of  development  and  most  outcome  variables. 
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Destabilizing  Influences 


Variable 

Timing  Implications 

1.  Reductions  in  project 
funding 

Potential  for  change  in  administration  every  48  months;  typical 
turn-over  in  key  military  leaders  occurs  every  24-36  months. 
Potential  change  in  key  Congress  positions  every  24  months; 
likelihood  increases  with  development  duration 

2.  Uncertainty  in  project 
funding 

Potential  for  change  in  administration  every  48  months;  typical 
turn-over  in  key  military  leaders  occurs  every  24-36  months.  . 
Potential  change  in  key  Congress  positions  every  24  months; 
likelihood  increases  with  development  duration. 

3.  Change  in  system 
requirements 

Changes  in  the  threat  environment  occur  unpredictably,  but 
become  more  likely  with  longer  development  durations. 

Changes  in  doctrine  and  system  requirements  follow  a  similar 
pattern. 

4.  Change  in  key  user 
representatives 

Typical  turn-over  in  such  key  military  positions  occurs  every 
—36  months 

5.  Change  in  key  project  team 
members 

Typical  turn-over  in  military  acquisition  positions  occurs  every 
-36  months.  Longer  development  durations  present  more 
opportunities  for  career  moves  on  the  part  of  key  civilian  team 
members 
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Central  Conclusion 


Shorter  development  cycle  times  favorably 
correlate  with  key  project  outcome  variables, 
largely  by  minimizing  the  exposure  of  the  project 
to  destabilizing  influences 


Length  of  Project  Development  and  Project  Performance 

(Average  number  of  successful  outcomes) 

Three  years 

Over  3  vears 

or  less 

Sig.  at 

Length  of  development  2.00 

4.71 

.002 
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